Critique of Human-powered Sorts & Joins

Marcus et al. (2011) describe using Amazon’s Mechanical Turk (MTurk) systems utilizing crowds for database sorts and joins. Crowdsourcing activities for these types of queries are typical costly in order to ensure accuracy and reliability. The authors present an interface for optimizing these functions in order to determine cost-effectiveness of various methods (simple join, naïve batching, and smart batching). Simple batching presents the MTurk responder, the one agreeing to respond to the request, with a simple interface with a dichotomous yes or no question regarding similarity of images. Naïve batching is similar in nature in that responders are presented with a dichotomous yes or no question, but there are multiple images presented in pairs to which the responder must answer. Smart batching presents the responder with several images to which they must draw connecting lines between images that are similar. Smart batching was quite successful in terms of both accuracy and cost reduction from $67 to about $3.